Multiple interaction auction

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for distributing content items. In one aspect, a method includes identifying a content item that is eligible to be provided in response to a content item request and is associated with a bid. It is determined that the identified content item includes two or more different interactive elements that each cause different actions to be initiated in response to user interaction with the different interactive elements. An auction score is determined for the content item based on a function of the bid, a bid modifier for each different interaction, and a probability that each interaction will be invoked. The content item is selected to be provided based on the auction score. Data that cause presentation of the selected content item at a user device are output.

BACKGROUND

This specification relates to data processing and content distribution.

The Internet facilitates the exchange of information and transactionsbetween users across the globe. This exchange of information enablescontent sponsors to provide sponsored content to a variety of users. Acontent sponsor can control the distribution of their content items(e.g., promotions, advertisements, audio files, video files, or othercontent items) based on a set of distribution parameters that specifyunder what conditions a content item is eligible to be distributed. Whena presentation opportunity meeting the conditions is available, thecontent item provided by a content sponsor is deemed eligible to beprovided for presentation.

SUMMARY

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving a content item request; identifying, based on informationincluded in the content item request, a content item that is eligible tobe provided in response to the content item request and is associatedwith a bid specifying an amount that a content sponsor has specified fordistribution of the content item; determining that the identifiedcontent item includes two or more different interactive elements thateach cause different actions to be initiated in response to userinteraction with the different interactive elements; identifying, foreach different action, a probability that the action will be invokedthrough user interaction with an interactive element of the content itemand a bid modifier for the interaction; determining an auction score forthe content item based on a function of the bid, the bid modifier foreach different interaction, and the probability that each interactionwill be invoked; selecting, based on the auction score, the content itemto be provided in response to the content item request; and outputtingdata that cause presentation of the selected content item at a userdevice. Other embodiments of this aspect include corresponding systems,apparatus, and computer programs, configured to perform the actions ofthe methods, encoded on computer storage devices.

These and other embodiments can each optionally include one or more ofthe following features. Methods can include the action of determining abaseline cost for the content item based on a next highest auction scorerelative to the auction score for the content item, the probability thateach interaction will be invoked, and the bid modifier for eachinteraction.

Methods can include the actions of determining that one of the differentactions was invoked after the content item was provided for presentationat the user device; and determining a price of providing the contentitem based on the baseline cost and the bid modifier of the invokedaction.

Determining the price can include determining a product of the baselinecost and the bid modifier. Determining a baseline cost can includedetermining a ratio of a sum of the next highest auction score and anoverall negative effect of providing the content item relative to aweighted sum of the probabilities that each of the actions will beinvoked.

Determining the ratio can include determining a first product of a firstprobability that a user will invoke a first action and the bid modifierfor the first action; determining a second product of a secondprobability that a user will invoke a second action and the bid modifierfor the second action; summing the first product and the second productto obtain the weighted sum of the probabilities; summing the nexthighest auction score and the overall negative effect of providing thecontent item to obtain a modified next highest auction score; anddetermining the baseline cost based on a ratio of the modified nexthighest auction score and the weighted sum of the probabilities.

Determining an auction score for the content item can includedetermining a difference between an estimated benefit corresponding todistribution of the content item and an overall negative effect ofproviding the content item.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize none, one, or more ofthe following advantages. Allocation techniques described throughoutthis document provide a single auction score that can be used todistribute content items that each have multiple different, andpotentially disparate, actions that can be separately invoked throughdifferent user interactions with the content item. These techniques alsoenable a content distribution apparatus to determine different pricesthat will be paid by a content sponsor based, in part, on a single bidthat is provided by the content sponsor. The prices paid by the contentsponsor for the different actions can each reflect the benefit of theaction to the content sponsor, thereby facilitating more efficientdistribution of content items by a content sponsor. The techniquesdescribed enable direct comparison of content items in an auctionenvironment despite the fact that different content items may providedifferent actions that a user can invoke.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example environment in which content isdistributed to user devices.

FIG. 2 is a block diagram of an example data flow for selecting contentitems for distribution.

FIG. 3 is a flow chart of an example process for selecting a set ofcontent items for distribution.

FIG. 4 is a flow chart of an example process for determining a price ofdistributing a content item.

FIG. 5 is a block diagram of an example computer system.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

This document describes apparatus, systems, methods, and computerreadable medium related to selecting content items for distribution. Insome implementations, a selection process can select, for presentationin response to a content item request, a set of content items (e.g.,advertisements) that facilitate performance of various different actionsby a user to whom the content items are presented. For example, thecontent items can have various combinations of interaction elements thatenable users to invoke actions such as a click to call action, a maprequest action, a request for a landing page, or other actions.

As discussed in more detail throughout this document, a content sponsorthat provides a content item that enables a user to invoke multipledifferent actions may value each of those different actions differently.Additionally, each different content sponsor may value various differentactions differently. The different values attributed to each of thedifferent actions by the various different content sponsors, as well asthe variety of different actions that can be associated with thedifferent content items can make it difficult to directly comparecontent items for purposes of selecting content items that will beprovided in response to a particular content item requests.

The techniques discussed throughout this document facilitate directcomparison of content items despite the fact that each of the contentitems may provide a different set of actions, and despite the fact thatthe various content sponsors may attach a different value to the sameaction. In some implementations, the selection of the content items isfacilitated using an auction score that takes into account, at least inpart, the relative value of each action to a particular content sponsorand the probability that each of the actions will be invoked by a userif the content item is presented in response to the content itemrequest. The auction scores that are obtained for each of the contentitems can be used to select a set of content items that will be providedin response to the request (e.g., the content items having the highestauction scores).

When it is determined that one of the actions was invoked by a user, thecontent sponsor of the content item from which the action was invokedcan be charged a price for distribution of the content item. The pricethat the content sponsor is charged can be based on a minimum bidrequired for the content sponsor to maintain their ranking among thecontent items (e.g., based on the auction scores) and the bid modifierthat the content sponsor provided for the action that was invoked, asdescribed in detail with reference to FIG. 4. As described in moredetail below, a content sponsor can specify a different bid for eachdifferent action instead of specifying a single minimum bid and a bidmodifier for each action.

The following description refers to selection and distribution ofadvertisements, which are a type of content item. The description thatfollows is also applicable to other types of content items. For example,any video file, audio file, or other content item can be distributedusing the techniques described below.

FIG. 1 is a block diagram of an example environment 100 in which contentis distributed to user devices 106. The example environment 100 includesa network 102, such as a local area network (LAN), a wide area network(WAN), the Internet, or a combination thereof. The network 102 connectswebsites 104, user devices 106, content sponsors 108, and a contentdistribution system 110. The example environment 100 may include manydifferent websites 104, user devices 106, and content sponsors 108.

A website 104 is one or more resources 105 associated with a domain nameand hosted by one or more servers. An example website is a collection ofweb pages formatted in hypertext markup language (HTML) that can containtext, images, multimedia content, and programming elements, such asscripts. Each website 104 is maintained by a publisher, which is anentity that controls, manages and/or owns the website 104.

A resource 105 is any data that can be provided over the network 102. Aresource 105 is identified by a resource address that is associated withthe resource 105. Resources include HTML pages, word processingdocuments, and portable document format (PDF) documents, images, video,and feed sources, to name only a few. The resources can include content,such as words, phrases, images and sounds, that may include embeddedinformation (such as meta-information in hyperlinks) and/or embeddedinstructions (such as scripts). Units of content that are presented in(or with) resources are referred to as content items, and an individualcontent item can be stored in a single file or set of files independentof the resource.

A user device 106 is an electronic device that is capable of requestingand receiving resources over the network 102. Example user devices 106include personal computers, mobile communication devices, and otherdevices that can send and receive data over the network 102. A userdevice 106 typically includes a user application, such as a web browser,to facilitate the sending and receiving of data over the network 102.

A user device 106 can submit a resource request 112 that requests aresource 105 from a website 104. In turn, data representing therequested resource 114 can be provided to the user device 106 forpresentation by the user device 106. The requested resource 114 can be,for example, a home page of a website 104, a web page from a socialnetwork, or another resource 105. The data representing the requestedresource 114 can include data that cause presentation of resourcecontent 116 at the user device 106. The data representing the requestedresource 114 can also include data specifying a content item slot 118.

A content item slot is a portion of the resource (e.g., a portion of aweb page) or a portion of a user display (e.g., a presentation locationof another window or in a slot of a web page) in which one or morecontent items, such as advertisements, can be presented. A content itemsslot 118 can also be referred to as an advertisement slot, but any typeof content (e.g., content items other than advertisements) can bepresented in the content item slot 118.

A single content item slot 118 may be configured to include one or morepresentation positions 119 a and 119 b. Alternatively or additionally,each different content item slot 118 can be considered a separatepresentation position, and a resource can include multiple differentcontent item slots. Each presentation position can represent a portionof the content item slot 118 at which a content item can be presented.In some implementations, the number of presentation positions and/or thesize of the presentation positions for a particular content item slot118 may be determined based on the number, type, and/or value of contentitems that are available for presentation in the content item slot.

To facilitate searching of resources, the environment 100 can include asearch system 113 that identifies the resources by crawling and indexingthe resources provided by the publishers on the websites 104. Data aboutthe resources can be indexed based on the resource with which the dataare associated. The indexed and, optionally, cached copies of theresources are stored in a search index 122. Data that are associatedwith a resource is data that represents content included in the resourceand/or metadata for the resource.

User devices 106 can submit search queries to the search system 113 overthe network 102. In response, the search system 113 accesses the searchindex 122 to identify resources that are relevant to the search query.The search system 113 identifies the resources in the form of searchresults and returns the search results to the user device in searchresults page. A search result is data generated by the search system 113that identifies a resource that is responsive to a particular searchquery, and includes a link to the resource. An example search result caninclude a web page title, a snippet of text or a portion of an imageextracted from the web page, and the URL of the web page. Like otherresources, search results pages can include one or more content itemslots 118 in which content items, such as advertisements, can bepresented.

When a resource 105 is requested by a user device 106, execution of codeassociated with a content item slot 118 in the resource initiates arequest for one or more content items to populate the slot, which isreferred to as a content item request. The content item request caninclude characteristics of the slots that are defined for the requestedresource 114. For example, a reference (e.g., URL) to the requestedresource 114 for which the content item slot 118 is defined, a size ofthe content item slot, a maximum number of presentation positions (orcontent items) that can be included in the content item slot 118, and/ormedia types that are eligible for presentation in the content item slot118 can be provided to the content distribution system 110. Similarly,keywords associated with a requested resource (“resource keywords”) orentities that are referenced by the resource can also be provided to thecontent distribution system 110 to facilitate identification of contentitems that are relevant to the requested resource 114. Content itemrequests can also include other information, such as information thatthe user has provided, geographic information indicating a state orregion from which the request was submitted, or other information thatprovides context for the environment in which the content item will bedisplayed (e.g., a type of device at which the content item will bedisplayed, such as a mobile device or tablet device).

The content items that are provided in response to a content itemrequest (or another request) are identified based, in part, ondistribution parameters associated with the content items. Distributionparameters are a set of criteria upon which distribution of contentitems are conditioned. In some implementations, the distributionparameters for a particular content item can include distributionkeywords that must be matched (e.g., by resource keywords or searchqueries) in order for the content item to be eligible for presentation.The distribution parameters can also require that the content itemrequest include information specifying a particular geographic region(e.g., country or state) and/or information specifying that the contentitem request originated at a particular type of user device. Thedistribution parameters can also specify a bid and/or budget fordistributing the particular content item. As described in more detailbelow, bids can be used to select a set of content items forpresentation with a resource and/or to determine in which presentationposition the content item(s) will be presented.

The content items are selected for presentation in the presentationposition based on the outcome of a content item selection process. Insome implementations, the content item selection process can include anauction. The auction can be performed based, at least in part, based onbids that are associated with the content items. For example, assumethat a first content item is associated with a cost per click (CPC) bidof $1.25 and a second content item is associated with a CPC bid of$1.00. In this example, the first content item may be selected forpresentation in the presentation position 119 a (e.g., a highestpresentation position) and the second content item can be selected forpresentation in presentation position 119 b because the bid for contentitem 1 is higher than the bid for content item 2.

In the example above, the content items were selected based solely onthe values of their respective CPC bids. However, content items can beselected based on auction scores that are determined based on additional(or other) information. For example, an auction score for a particularcontent item can be a product of a CPC bid of the content item and aprobability that a user will click on the content item to request alanding page to which the content item links (e.g., CPC bid*p_click,where p_click is the probability that a user will click the content itemwhen presented). Note that for purposes of example, this document refersto bid generally, and that a bid can be specified as a CPC bid or a CPM(“cost-per-mille”) bid, and CPC bids can be converted to an eCPM(estimated CPM) value so that the bids can be directly compared.

The eCPM bid can be determined, for example, based on a product of aninteraction rate (e.g., click rate) for a content item and the CPC bidfor the content item. If the content item offers two or more differentactions that a user can invoke through interaction with the contentitem, the eCPM for that content item can be determined based on a sum ofthe products of the interaction rate for the action and the bid (e.g.,Σ₁ ^(x) InteractionRate_(x)*bid, where InteractionRate_(x) is theinteraction rate for Action_x and bid is the bid specified for thecontent item).

The probability that a particular content item will receive a click (oranother interaction) can be based on a historical click through rate (orinteraction rate) of the content item. For example, assume that aparticular content item has a historical click rate of 0.10, indicatingthat the particular content item receives one click for every tenpresentations of the content item. In this example, the probability thatthe content item will receive a click can be set to 0.10. Thus, thevalue 0.10 can be multiplied by the CPC bid for the particular contentitem to obtain the auction score for the particular content item. Inturn, the content items having the highest N auction scores can beselected for presentation in N available presentation positions, where Nis an integer number of available presentation positions.

Some content items provide the ability for users to take actions beyondthe user clicking on the advertisement to request an advertiser landingpage. For example, as illustrated by the example advertisement 119 a, acontent item can provide actions such as a headline click action inwhich user interaction with a headline 150 (e.g., advertisement text oranother portion of the advertisement that does not invoke anotheraction), referred to as a headline click, causes a user device torequest a landing page associated with the example advertisement 119 a.The example advertisement 119 a can also provide a “click to call”action in which user interaction with a click to call interactiveelement 152 causes a user device (e.g., a mobile device or a device thatcommunicates using Internet Protocol) to call a phone numbercorresponding to the content item (e.g., an advertiser's call center orretail location). Similarly, the example advertisement 119 a can providea “get directions” or “map request” action in which user interactionwith a map interactive element 154 causes the user device to request aresource that presents a map and/or directions to an addresscorresponding to the content item (e.g., in a map interface).

The different actions a content item can provide may provide differentbenefits (e.g., value) to a content sponsor and/or a differentexperience for users depending on which action is invoked by the user.For example, as discussed in more detail throughout this document, aheadline click by a user may provide one value to the advertiser andprovide a specific experience for users, while a call to click actionmay provide a different value to the advertiser and provide a differentexperience for the user (e.g., talking to a sales representative or acustomer service agent), and a get directions action may provide someother value to the content sponsor and/or provide another differentexperience for the user (e.g., determining how to get to a retaillocation of the content sponsor). Thus, content sponsors may be willingto pay a different amount for each of the different actions that areperformed.

When content items are selected for presentation, it is not knownwhether a particular presentation of the content item will result in auser invoking zero, one, or more of the actions that can be invokedthrough user interaction with the different interaction elements thatare provided in a single content item. Additionally, if differentcontent items provide different actions, it can be difficult to directlycompare these content items when selecting which content items will beprovided in response to a content item request.

The environment 100 includes a content item selection apparatus (CISA)120 that facilitates selection of content items to be provided inresponse to a content item request. In some implementations, the CISA120 determines, for each of multiple different content items, an auctionscore (or selection score) that facilitates selection of content itemsirrespective of whether the content items provide the same actions, andwithout knowing which actions, if any, will actually be invoked by auser that is presented one or more of the content items.

As described in more detail below, the CISA 120 can determine theauction score for each content item based on a mathematical combination(e.g., function) of the bid for the content item, probabilities thateach of the different actions will be invoked if the content item ispresented to a user, and an estimated value to a user that invokes eachof the different actions. As discussed in more detail below, theestimated value to a user that invokes a particular action can be based,at least in part, the positive effect (e.g., positive value to the userand/or the CISA 120) of a good user experience when the particularaction is invoked and the negative effect (e.g., a negative value to theuser and/or the CISA 120) of a bad user experience when the particularaction is invoked. The combination of the estimated values of thevarious actions can be combined to provide an auction score that can becompared with auction scores of other advertisement (irrespective of howmany and/or which actions are provided by the various content items) toselect content items that will be provided in response to a content itemrequest.

FIG. 2 is a block diagram of an example data flow 200 for selectingcontent items for distribution. The data flow 200 begins with the CISA120 receiving a content item request 202. The content item request 202is a request for one or more content items (e.g., advertisements) thatwill be presented with a resource (e.g., web page or video). Forexample, as illustrated by FIG. 2, the content item request 202(“request”) can be submitted by a user device 106 when the resource 114is rendered at the user device 106. For example, execution of codeincluded in the resource 114 can cause the user device 106 to submit thecontent item request 202. Alternatively, the request 202 could beinitiated when playback of a video reaches a portion of the video atwhich advertisements will be inserted.

The CISA 120 can receive the content item request 202 directly from theuser device 106 or from another device or system (e.g., the contentdistribution system 110). For example, if the CISA 120 is implementedindependent of the content distribution system 110, the user device 106may submit the content item request 202 to the content distributionsystem 110, which can provide the content item request 202 (or dataassociated with the content item request 202) to the CISA 120.

The content item request 202 can include data specifying a number ofcontent items that can be presented with the resource 114. For example,as illustrated by FIG. 2, the resource 114 includes a content item slot118 having two presentation positions 119 a and 119 b. In this example,the content item request 202 may specify that up to two content itemscan be presented with the resource 114 (e.g., in presentation positions119 a and 119 b).

In response to receiving the request 202, the CISA 120 can identify aset of eligible content items 204 that are eligible to be provided inresponse to the request. The set of eligible content items can beidentified, as described for example with reference to FIG. 1, based ondistribution parameters associated with the content items and/orinformation included in the request 202. In some implementations, theset of eligible content items will include a set of a first content item(“CI_1”) 206, a second content item “(CI_2”) 208, and a third contentitem (“CI_3”) 210.

In some implementations, the set of eligible content items 204 areidentified by content item data 210 that is accessed from a content itemdata store 212. The content item data 210 can specify, for example, aname that identifies the content item, data specifying the type ofcontent item (e.g., text advertisement, image advertisement, or videoadvertisement), a bid specifying an amount that a content sponsor willpay for distribution of the content item, and other information relatedto the content item (e.g., dimensions, distribution parameters for thecontent item, or other information).

The content item data 210 can include an independent bid (e.g., separateand/or different bid) for each of the content items in the set ofcontent items 204. For example, the content item data 210 can specifythat the content item CI_1 has a bid of B1, the content item CI_2 has abid of B2, and the content item CI_3 has a bid of B3. The respectivevalues of B1, B2, and B3 can be specified by the content sponsors of therespective content items CI_1, CI_2, and CI_3.

The CISA 120 determines based, at least in part, on the content itemdata 210 a set of actions 214 that are made available by the set ofcontent items 204. For example, as illustrated by the table 216, theCISA 120 can determine that content item CI_1 provides the ability for auser to invoke Action_11 and Action_12 through interaction withinteraction elements of the content item CI_1, content item CI_2provides the ability for a user to invoke Action_21 and Action_22through interaction with content item elements of the content item CI_2,and content item CI_3 provides the ability for a user to invokeAction_31 and Action_32 through interaction with interaction elements ofthe content item CI_3.

As described above with reference to FIG. 1, the actions that can beinvoked through user interaction with interaction elements of a contentitem can include placing a call to a phone number referenced by thecontent item, requesting a map showing a location and/or directions to aretail location (or another location) referenced by the content item,requesting a landing page referenced by the content item, or otheractions (e.g., sending an e-mail to (or otherwise sharing informationwith) a specified set of users, posting information about the contentitem on a social network page (or stream) of the user, printing a couponor placing an order with the content sponsor).

The CISA 120 identifies, for each action in the set of actions 214, aprobability that a user will invoke the action when the content item ispresented to the user. For example, as illustrated by Table 218, theCISA 120 can determine that there is a probability of p11 that a userwill invoke Action_11 if CI_1 is presented, and a probability of p12that the user will invoke Action_12 if CI_1 is presented. Similarly, theCISA 120 can determine that if CI_2 is presented, there is a probabilityof p21 that the user will invoke Action_21 and a probability of p22 thatthe user will invoke Action_22. The CISA 120 can also determine that ifCI_3 is presented, there is a probability of p31 that the user willinvoke Action_31 and a probability of p32 that the user will invokeAction_32.

The probability that a user will invoke each particular action can beidentified, for example, from the content item data 210 and/or contentitem data store 212. For example, each action each content item can bestored with information specifying the probability that the user willinvoke each of the various actions that are provided by the contentitem.

In some implementations, the probability that a user will invoke aparticular action of a particular content item can be determined basedon historical data regarding user interactions with the particularcontent item. For example, the historical data, such as a log specifyingcontextual data for each previous impression (e.g., a resource withwhich the content item was presented, a time of day when the contentitem was presented, distribution criteria of the content item that werematched by the previous content item requests in response to which thecontent item was provided) and interaction data specifying which, ifany, actions were invoked by a user when the content item was presentedcan be input to a predictive model that outputs a probability (orlikelihood) that a user will invoke each of the actions provided by thecontent item. In some implementations, contextual data related to thereceived content item request 202 can also be input to the predictivemodel for purposes of obtaining probabilities that a user receiving thecontent item in response to the received content item request 202 willinvoke the actions offered by the set of content items 204. Theprobabilities output by the model can be stored in the content item datastore 212, and made available to the CISA 120.

Using at least the bids and the probabilities, the CISA 120 determines aset of auction scores 220 for the set of content items 204. For example,using the bid B1, and the probabilities P11 and P12, the CISA 120 candetermine an auction score AS_1 222 for the content item CI_1.Similarly, the CISA 120 can use the bid B2, the probabilities P21 andP22 to determine an auction score AS_2 224 for the content item CI_2,and also use the bid B3, the probabilities P31 and P32 to determine anauction score AS_3 226 for the content item CI_3.

In some implementations, the CISA 120 can determine an auction score foreach content item based on relationship (1).

$\begin{matrix}{{AS}_{i} = {\sum\limits_{1}^{x}\; {{pAction}_{x}*{bid}_{i}}}} & (1)\end{matrix}$

where,

AS_(i) is the auction score for content item i;

-   -   pAction_(x) is the probability that a user will invoke Action_x        that is provided by content item i if content item i is        presented; and    -   bid_(i) is the bid specified for the content item i.

Generally, an auction score provided by relationship (1) provides anestimated value to a content sponsor of a presentation of a contentitem. For example, the individual value of each interaction is providedas a product of the probability (or likelihood) that the user willinvoke the action and the bid of the content item.

Relationship (1) generally assumes that the content sponsor values eachdifferent action equally. However, as discussed above, each differentaction may provide a content sponsor a different amount of value. Forexample, assume that based on historical information, a content sponsordetermines that a click to call action provides 20% (or some otheramount) more value than a headline click. Further assume that thecontent sponsor determines that a map request action provides 10% (orsome other amount) more value than a headline click. In this example,the amount that the content sponsor values, and/or is willing to payfor, a click to call action may be 20% more than the content sponsor iswilling to pay for a headline click. Similarly, the content sponsor maybe willing to pay 10% more for a map request than a headline click ifthe content sponsor values a map request 10% more than a headline click.

In some implementations, a content sponsor can use bid modifiers (ordifferent bids) to specify different values that the content sponsorplaces on the different actions. A bid modifier is a value that is usedto adjust the bid that the content sponsor specifies for distribution ofa content item. The bid modifiers can be used to determine differentamounts that the content sponsor is willing to pay for the variousactions that are provided by the content items distributed for thecontent sponsor.

The content sponsor can specify a different bid modifier for each of thedifferent actions that are provided by a content item. Continuing withthe example above, the content sponsor can specify a bid modifier of1.20 (or another value) for the click to call action and a bid modifierof 1.10 for the map request action. The bid modifier of 1.20 indicatesthat the content sponsor values the click to call action more than theheadline click and is willing to pay 1.20 times the bid value when auser invokes the click to call action. Similarly, the bid modifier of1.10 indicates that the content sponsor values the map request actionmore than the headline click action, and is willing to pay 1.10 timesthe bid value when a user invokes the map request action. Generally, thebid specified by the content sponsor is a value that the content sponsoris willing to pay for a headline click, but the bid could be associatedwith a different action, and the content sponsor could also specify abid modifier for headline clicks if desired.

In some implementations, a content sponsor can specify different bidsfor each different action instead of (or in addition to) specifying asingle bid and different bid modifiers for each action. In the exampleabove, the content sponsor could specify a headline click bid of $1.00,a click to call bid of $1.20, and a map request bid of $1.10. This wouldenable the content sponsor to similarly differentiate the value that thecontent sponsor places on each of the different actions.

The bid modifiers (or different bids) specified by a content sponsor canbe stored in the content item data store 212. For example, each bidmodifier (or different bid) can be stored with a reference to thecontent item and action corresponding to the bid modifier (or differentbid). Thus, when the CISA 120 obtains the content item data 210 from thecontent item data store, the bid modifiers (or bids) can be included inthe content item data 210, and the CISA 120 can identify the bidmodifiers from the content item data 210 that are received.

In some implementations, the CISA 120 can use the bid modifiers (orbids) when determining the set of auction scores 220. For example, theCISA 120 can use relationship (2) to determine auction scores for thecontent items.

$\begin{matrix}{{AS}_{i} = {\sum\limits_{1}^{x}\; {{pAction}_{x}*{bid}_{i}*m_{x}}}} & (2)\end{matrix}$

where,

AS_(i) is the auction score for content item i;

-   -   pAction_(x) is the probability that a user will invoke Action_x        that is provided by content item i if content item i is        presented;

bid_(i) is the bid specified for the content item i; and

m_(x) is the bid modifier for Action_x of content item i.

The auction score obtained using relationship (2) considers thedifferent values that the content sponsor has attached to the variousactions that a user can invoke using the content item (e.g., using thebid modifier). Thus, relationship (2) provides an auction score that isbased, in part, on the different benefit of each action to the contentsponsor.

When a content sponsor specifies different bids for the differentactions, mx of relationship (2) (and all other relationships discussedin this document) can be determined as a ratio of the bid for theparticular action relative to a reference bid, such as the headline bid(e.g., bid_(s)/bid_(r), where bid_(s) is the bid for the action and bid,is the reference bid).

A user that invokes the click to call action (or another action) mayhave a positive experience during the call (e.g., satisfied with thecustomer service or outcome of the call), but it is possible that theuser has a negative experience during the call (e.g., dissatisfied withcustomer service provided or outcome of the call). If the user has anegative experience, the user may attribute the negative experience tothe CISA 120 that provided the content item.

In some implementations, the auction scores determined by the CISA 120can take into account the possibility that the user will have a negativeexperience, and the impact that the negative experience may have on theuser's attitude towards content items provided by the CISA 120. Forexample, the CISA 120 can use relationship (3) to determine the set ofauction scores 220.

$\begin{matrix}{{AS}_{i} = {{\sum\limits_{1}^{x}\; \left\lbrack {{pAction}_{x}*{bid}_{i}*m_{x}} \right\rbrack} - {\sum\limits_{1}^{x}\; {{BadVisitCost}_{x}*{pBadVisit}_{x}*{pAction}_{x}}}}} & (3)\end{matrix}$

where,

AS_(i) is the auction score for content item i;

-   -   pAction_(x) is the probability that a user will invoke Action_x        that is provided by content item i if content item i is        presented;    -   bid_(i) is the bid specified for the content item i;    -   m_(x) is the bid modifier for Action_x of content item i;    -   BadVisitCost_(x) is a negative effect (e.g., expressed in a        monetary value or some other value) of a bad user experience        when Action_x of content item i is invoked;

pBadVisit, is the probability that a user will have a bad experienceafter invoking Action_x of content item i; and

Σ₁ ^(x) BadVisitCost_(x)*pBadVisit_(x)*pAction_(x) is an expression ofan overall negative effect of providing the content item.

The BadVisitCost, can be specified by an administrator of the CISA 120.For example, if an administrator of the CISA 120 has determined the costof a bad user experience for each of the actions, the CISA 120 canreceive those values as input from the administrator of the CISA 120. Insome implementations, the value assigned to BadVisitCost, can beestimated based on the relative impact that each type of action has on auser. For example, a bad user experience during a phone call may begenerally more detrimental (e.g., expensive) than a bad user experiencewith a map that is provided in response to a map request, and a bad userexperience with the map may be generally more detrimental (e.g.,expensive) than a bad user experience with a headline click. Thus, theBadVisitCost for the click to call action may be set higher than theBadVisitCost for a map request action, which can be set higher than theBadVisitCost for a headline click. The BadVisitCost for each action canbe stored, for example, in the content item data store 212 and/orreceived with the content item data 210.

In some implementations, the BadVisitCost can be specified on aper-query basis. For example, a particular BadVisitCost constant can bespecified on a global basis (e.g., for all queries), for each particularcategory of queries (e.g., automobiles, art, or sports), or for eachparticular query. When the BadVisitCost is specified as a constant, itcan either be stored in the content item data store 212, or bemaintained in the CISA 120.

The probability specified by pBadVisit_(x) can be determined based onuser feedback (e.g., survey information) and/or inferred based onprevious actions of users. For example, user survey information mayreveal the portions of users that have bad experiences after invokingeach of the different actions. In a particular example, the survey datamay reveal that 5% of the users that invoke the click to call actionhave a bad experience (e.g., not connected, long hold time,dissatisfaction with outcome), while 3% of the users that invoke the maprequest action have a bad experience (e.g., map fails to load ordirections are inaccurate), and 1% of the users that perform a headlineclick have a bad experience (e.g., failed page load or dissatisfactionwith landing page content). In this example, the pBadVisit for the clickto call, map request, and headline click can be respectively set to0.05, 0.03, and 0.01. These values can be stored in the content itemdata store 212 and/or received by the CISA 120 with the content itemdata 210.

Using the set of auction scores 220, the CISA 120 ranks the set ofcontent items 204. For example, assuming that AS_1>AS_2>AS_3, the CISA120 can rank the set of content items 204 in the following orderCI_1>CI_2>CI_3.

Using the rankings, the CISA 120 can select a set of winning contentitems 228 to be provided in response to the content item request 202. Insome implementations, the set of winning content items 228 includes theN content items having the highest N auction scores, where N is a numberof content items to be provided in response to the content item request202. In the present example, the resource 114 includes two presentationpositions 119 a and 119 b. Assuming that a content item will bepresented in both content item slots 119 a and 119 b, the set of winningcontent items 228 in the present example will include CI_1, which hasthe highest auction score AS_1, and CI_2, which has the second highestauction score AS_2.

The CISA 120 outputs the set of winning content items 228 forpresentation at the user device 106 in response to the content itemrequest 202. In some implementations, the CISA 120 outputs the set ofwinning content items 228 by generating and sending code that cause theuser device 106 to display the set of winning content items. Forexample, the CISA 120 can either transmit the set of wining contentitems (e.g., the files of the winning content items) to the user device106, or transmit code that instructs the user device to fetch the set ofwinning content items 228 from one or more network locations.

The CISA 120 can also determine a price 230 that will be paid fordistribution of each content item in the set of winning content items228. The determination of a price to be paid for distribution of the setof winning content items 228 is discussed in detail with reference toFIG. 4.

FIG. 3 is a flow chart of an example process 300 for selecting a set ofcontent items for distribution. The process 300 can be performed, forexample, by one or more data processing apparatus, such as the CISA 120and/or the content distribution system 110 of FIG. 1. Operations of theprocess 300 can be implemented by instructions that when executed causeone or more data processing apparatus to perform operations of theprocess 300. The instructions can be stored on a non-transitory computerreadable medium.

A request for one or more content items is received (302). In someimplementations, the content item request is a request for one or morecontent items to be presented in at least some of two or morepresentation positions of a resource. As discussed above with referenceto FIG. 1, a content item can include multiple presentation positions.The content item request can include information specifying the numberof presentation positions, which can correspond to a maximum number ofcontent items that can be presented in a single page view and/orprovided in response to the request.

A set of eligible content items are identified (304). In someimplementations, the set of eligible content items are identified basedon information included in the content item request. For example, asdiscussed above with reference to FIG. 1, each eligible content item canbe identified based on the eligible content item having distributionparameters that are matched by information included in the content itemrequest.

In some implementations, the content items in the identified set ofeligible content items are associated with a bid. The bid for eachcontent item specifies, for example, an amount that a content sponsorfor the content item has specified for distribution of the content item.For example, a particular content sponsor may specify a CPC bid of $1.00indicating that the content sponsor is willing to pay up to $1.00 foreach headline click that the content item receives. In someimplementations, CPC bids can be converted to eCPM bids as discussedabove with reference to FIG. 1.

A determination is made that at least one of the eligible content itemsprovide users the ability to invoke two or more different actions (306).In some implementations, the determination that a content item providesa user the ability to invoke two or more different actions includes adetermination that the content item includes two or more differentinteractive elements that each cause different actions to be initiatedin response to user interaction with the different interactive elements.

For example, a determination can be made that a particular content itemincludes a first interactive element that requests a particular landingpage in response to user interaction with the first interactive element.Similarly, a determination can be made that the particular content itemalso includes a second interactive element that causes a user device toplace a phone call to a specified phone number in response to userinteraction with the second interactive element. Other interactiveelements that users can interact with to invoke other actions can alsobe determined to be included in the particular content item.

In some implementations, the determination that the particular contentitem includes multiple different interaction elements and/or providesthe ability for a user to invoke multiple different actions can be madebased on information provided by a content sponsor of the particularcontent item. For example, the content sponsor may specify the differentactions that can be invoked through user interaction with the particularcontent item. The data specifying the different actions can be stored,for example, in a data store and accessed when the particular contentitem is identified as an eligible content item for a content itemrequest.

A probability that each different action will be invoked is identified(308). In some implementations, the probability that each differentaction will be invoked is based, for example, on a probability that auser will interact with the interaction element of the content item thatcauses the action to be performed. For example, if the content itemincludes a click to call interaction element, the probability that theclick to call action will be invoked by a user can be based on thelikelihood that the user will interact with the click to callinteraction element. The probability that each different action will beinvoked can be accessed, for example, from a data store that includesstored associations between content items, actions, and probabilitiesthat each of the actions provided by each of the content items will beinvoked when the content item is presented. The probability that eachaction will be invoked can be determined, for example, based onhistorical data as discussed above with reference to FIG. 2.

One or more bid modifiers are identified for the different actions(310). As discussed above with reference to FIG. 2, each bid modifier isa value (e.g., a multiplier or divisor) that is used to adjust the bidthat the content sponsor specifies for a content item. In someimplementations, a content sponsor can specify a different bid modifierfor zero, one, or more of the different actions that are provided by thecontent items. For example, as discussed above with reference to FIG. 2,a content sponsor can specify higher bid modifiers for actions thatprovide greater benefit (e.g., monetary value or branding value) thanthe bid modifiers (if any) that are assigned to other actions thatprovide less benefit. The bid modifiers can be identified (or accessed)from a data store (e.g., data store 212 of FIG. 2) that storesassociations between content items, actions provided by the contentitems, and bid modifiers for the actions.

An auction score is determined for each of the eligible content items(312). In some implementations, the auction score is determined based ona function of the bid, the bid modifier for each different action, andthe probability that each interaction will be invoked by a user if thecontent item is provided in response to the content item request. Theauction score can be determined, for example, based on a differencebetween a sum of benefits (e.g., positive values to the user and/or CISA120) and a sum of costs (e.g., negative values to the user and/or CISA120) associated with presentation of the content item.

For example, as discussed above with reference to FIG. 2 andrelationships (1)-(3), the auction score for each content item can be ascore specifying a value associated with presentation of the contentitem, which takes into account the probability that a user will invokeeach action, the bid modifiers representing the perceived or estimatedvalue of each action relative to the perceived or estimated values ofthe other actions, a probability that the user has a bad experienceafter invoking each action, and the cost to the CISA 120 of the userhaving a bad experience after invoking each of the actions. As discussedabove, the auction score can be determined using any of therelationships (1)-(3).

A set of content items to be provided in response to the request isselected (314). In some implementations, the set of content items can beselected based on the auction scores for the eligible content items. Forexample, as discussed above with reference to FIG. 2, the set of contentitems can include the N content items that have the highest N auctionscores, where N is the number of content items that will be provided inresponse to the content item request.

Data that cause presentation of the set of content items are generatedand provided (316). In some implementations, the data include datarepresenting each of the selected content items. In someimplementations, the data include code that causes the user device toretrieve data representing the content items are provided to the userdevice. For example, the code can include instructions that cause theuser device to retrieve the content items from a specified networklocation (e.g., from a specified content item repository).

FIG. 4 is a flow chart of an example process for determining a price ofdistributing a content item. The process 400 can be performed, forexample, by one or more data processing apparatus, such as the CISA 120and/or the content distribution system 110 of FIG. 1. Operations of theprocess 400 can be implemented by instructions that when executed causeone or more data processing apparatus to perform operations of theprocess 400. The instructions can be stored on a non-transitory computerreadable medium.

A baseline cost for distribution of the content item is determined(402). In some implementations, the baseline cost is an amount that acontent sponsor will pay for distribution of a content item when anaction that does not have a bid modifier, or has a bid modifier (e.g.,multiplier) of 1.0 is invoked. For example, assume that a contentsponsor does not specify a bid modifier for a headline click (orspecifies a bid modifier of 1.0). Further assume that the content itemreceives a headline click when presented in response to a content itemrequest. In this example, the amount paid by the content sponsor fordistribution of the content item can be the baseline cost.

In some implementations, the baseline cost is determined based, at leastin part, on a next highest auction score. For example, with reference toFIG. 2, the baseline cost of distributing CI_1, which is associated withthe highest auction score AS_1, can be determined based, in part, on theauction score AS_2, which is the second highest auction score in thatexample. Similarly, the baseline cost of distributing CI_2, which isassociated with the second highest auction score AS_2, can be determinedbased, in part, on the third highest auction score AS_3. As discussed inmore detail below, the baseline cost can also be determined based on thebid modifiers for the actions provided by the content item, and theprobability the each of the actions will be invoked if the content itemis provided in response to the content item request.

The baseline cost can be, for example, the minimum amount that a contentsponsor must pay in order to maintain their ranking in the content itemselection process. For example, the baseline cost for CI_1 in theexample above can be the minimum amount that the content sponsor of CI_1must pay (or bid) to maintain the highest auction score. In someimplementations, the minimum amount that a content sponsor must pay inorder to maintain their ranking can be determined by determining the bidthat will keep the auction score for the content item at least equal tothat of the next highest auction score. The minimum amount can bedetermined, for example, based on a ratio of a sum of the next highestauction score and an overall negative effect (e.g., to the CISA 120) ofproviding the content item relative to a weighted sum of theprobabilities that each of the actions will be invoked, as representedby relationship (4).

$\begin{matrix}{{BaselineCost}_{i} = \frac{{NextHighestAS} + {OverallNegativeEffect}_{i}}{\sum\limits_{1}^{x}\; {m_{x}*{pAction}_{x}}}} & (4)\end{matrix}$

where,

BaselineCost_(i) is the baseline cost for content item i;

NextHighestAS is the next highest auction score relative to the auctionscore for content item i;

OverallNegativeEffect_(i) is the overall negative effect associated withpresentation of content item i;

pAction_(x) is the probability that a user will invoke Action_x of thecontent item i; and

m_(x) is the bid modifier for Action_x of content item i.

In some implementations, the OverallNegativeEffecti can be determinedusing relationship (5).

$\begin{matrix}{\sum\limits_{1}^{x}\; {{BadVisitCost}_{x}*{pBadVisit}_{x}*{pAction}_{x}}} & (5)\end{matrix}$

where,

BadVisitCost_(x) is a negative effect (e.g., expressed in a monetaryvalue) of a bad user experience when Action_x of content item i isinvoked;

pBadVisit_(x) is the probability that a user will have a bad experienceafter invoking Action_x of content item i; and

pAction_(x) is the probability that a user will invoke Action_x that isprovided by content item i if content item i is presented.

According to relationships (4) and (5), the baseline cost can bedetermined, in part, by determining, for each of the actions, a weightedprobability (e.g., a product of the probability that a user will invokethe action and a bid modifier for the action). The weightedprobabilities determined for each of the actions are summed to obtain aweighted sum of the probabilities. The next highest auction score issummed with the overall negative effect of providing the content item toobtain a modified next highest auction score. In turn, the baseline costis determined based on the ratio of the modified next highest auctionscore relative to the weighted sum of the probabilities.

A determination is made that one of the different actions was invokedafter presentation of the content items at the user device (404). Insome implementations, the determination that one of the differentactions was invoked is made based on receipt of data from the userdevice indicating that the user interacted with an interface elementthat causes the action to be performed. For example, user interactionwith a click to call interface element can cause the user device tosubmit interaction data to the CISA 120 or the content distributionapparatus 110 (or another processing apparatus) indicating that the userinteracted with the click to call interface element.

A price of providing the content item is determined in response todetermining that one of the different actions was invoked (406). In someimplementations, the price of providing (or distributing) the contentitem can be based on the baseline cost and the bid modifier of theinvoked action. For example, the price of providing the content item canbe a product of the baseline cost and the bid modifier of the invokedaction. For purposes of illustration, assume that a user invokedAction_11 after presentation of CI_1. Also assume that the baseline costfor CI_1 is BC_1 and that the content sponsor of CI_1 specified a bidmodifier of m_11 for the Action_11 of content item CI_1. In thisexample, price of providing content item CI_1 can be the result ofBC_1*m_11. The content sponsor of CI_1 can be charged this price fordistribution of the content item CI_1.

FIG. 5 is a block diagram of an example computer system 500 that can beused to perform operations described above. The system 500 includes aprocessor 510, a memory 520, a storage device 530, and an input/outputdevice 540. Each of the components 510, 520, 530, and 540 can beinterconnected, for example, to the CISA 120 using a system bus 550. Theprocessor 510 is capable of processing instructions for execution withinthe system 500. In one implementation, the processor 510 is asingle-threaded processor. In another implementation, the processor 510is a multi-threaded processor. The processor 510 is capable ofprocessing instructions stored in the memory 520 or on the storagedevice 530.

The memory 520 stores information within the system 500. In oneimplementation, the memory 520 is a computer-readable medium. In oneimplementation, the memory 520 is a volatile memory unit. In anotherimplementation, the memory 520 is a non-volatile memory unit.

The storage device 530 is capable of providing mass storage for thesystem 500. In one implementation, the storage device 530 is acomputer-readable medium. In various different implementations, thestorage device 530 can include, for example, a hard disk device, anoptical disk device, a storage device that is shared over a network bymultiple computing devices (e.g., a cloud storage device), or some otherlarge capacity storage device.

The input/output device 540 provides input/output operations for thesystem 500. In one implementation, the input/output device 540 caninclude one or more of a network interface devices, e.g., an Ethernetcard, a serial communication device, e.g., and RS-232 port, and/or awireless interface device, e.g., and 802.11 card. In anotherimplementation, the input/output device can include driver devicesconfigured to receive input data and send output data to otherinput/output devices, e.g., keyboard, printer and display devices 560.Other implementations, however, can also be used, such as mobilecomputing devices, mobile communication devices, set-top box televisionclient devices, etc.

Although an example processing system has been described in FIG. 5,implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in other types ofdigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A method performed by data processing apparatus,the method comprising: receiving a content item request; identifying,based on information included in the content item request, a contentitem that is eligible to be provided in response to the content itemrequest and is associated with a bid specifying an amount that a contentsponsor has specified for distribution of the content item; determining,by one or more data processing apparatus, that the identified contentitem includes two or more different interactive elements that each causedifferent actions to be initiated in response to user interaction withthe different interactive elements; identifying, for each differentaction, a probability that the action will be invoked through userinteraction with an interactive element of the content item and a bidmodifier for the interaction; determining, by one or more dataprocessing apparatus, an auction score for the content item based on afunction of the bid, the bid modifier for each different interaction,and the probability that each interaction will be invoked; selecting,based on the auction score, the content item to be provided in responseto the content item request; and outputting, by one or more dataprocessing apparatus, data that cause presentation of the selectedcontent item at a user device.
 2. The method of claim 1, furthercomprising: determining a baseline cost for the content item based on anext highest auction score relative to the auction score for the contentitem, the probability that each interaction will be invoked, and the bidmodifier for each interaction.
 3. The method of claim 2, furthercomprising: determining that one of the different actions was invokedafter the content item was provided for presentation at the user device;and determining a price of providing the content item based on thebaseline cost and the bid modifier of the invoked action.
 4. The methodof claim 3, wherein determining the price comprises determining aproduct of the baseline cost and the bid modifier.
 5. The method ofclaim 2, wherein determining a baseline cost comprises determining aratio of a sum of the next highest auction score and an overall negativeeffect of providing the content item relative to a weighted sum of theprobabilities that each of the actions will be invoked.
 6. The method ofclaim 5, wherein determining the ratio comprises: determining a firstproduct of a first probability that a user will invoke a first actionand the bid modifier for the first action; determining a second productof a second probability that a user will invoke a second action and thebid modifier for the second action; summing the first product and thesecond product to obtain the weighted sum of the probabilities; summingthe next highest auction score and the overall negative effect ofproviding the content item to obtain a modified next highest auctionscore; and determining the baseline cost based on a ratio of themodified next highest auction score and the weighted sum of theprobabilities.
 7. The method of claim 1, wherein determining an auctionscore for the content item comprises determining a difference between anestimated benefit corresponding to distribution of the content item andan overall negative effect of providing the content item.
 8. Anon-transitory computer storage medium encoded with a computer program,the program comprising instructions that when executed by one or moredata processing apparatus cause the one or more data processingapparatus to perform operations comprising: receiving a content itemrequest; identifying, based on information included in the content itemrequest, a content item that is eligible to be provided in response tothe content item request and is associated with a bid specifying anamount that a content sponsor has specified for distribution of thecontent item; determining that the identified content item includes twoor more different interactive elements that each cause different actionsto be initiated in response to user interaction with the differentinteractive elements; identifying, for each different action, aprobability that the action will be invoked through user interactionwith an interactive element of the content item and a bid modifier forthe interaction; determining an auction score for the content item basedon a function of the bid, the bid modifier for each differentinteraction, and the probability that each interaction will be invoked;selecting, based on the auction score, the content item to be providedin response to the content item request; and outputting data that causepresentation of the selected content item at a user device.
 9. Thecomputer storage medium of claim 8, wherein the instructions cause theone or more data processing apparatus to perform operations furthercomprising: determining a baseline cost for the content item based on anext highest auction score relative to the auction score for the contentitem, the probability that each interaction will be invoked, and the bidmodifier for each interaction.
 10. The computer storage medium of claim9, wherein the instructions cause the one or more data processingapparatus to perform operations further comprising: determining that oneof the different actions was invoked after the content item was providedfor presentation at the user device; and determining a price ofproviding the content item based on the baseline cost and the bidmodifier of the invoked action.
 11. The computer storage medium of claim10, wherein determining the price comprises determining a product of thebaseline cost and the bid modifier.
 12. The computer storage medium ofclaim 9, wherein determining a baseline cost comprises determining aratio of a sum of the next highest auction score and an overall negativeeffect of providing the content item relative to a weighted sum of theprobabilities that each of the actions will be invoked.
 13. The computerstorage medium of claim 12, wherein determining the ratio comprises:determining a first product of a first probability that a user willinvoke a first action and the bid modifier for the first action;determining a second product of a second probability that a user willinvoke a second action and the bid modifier for the second action;summing the first product and the second product to obtain the weightedsum of the probabilities; summing the next highest auction score and theoverall negative effect of providing the content item to obtain amodified next highest auction score; and determining the baseline costbased on a ratio of the modified next highest auction score and theweighted sum of the probabilities.
 14. A system comprising: a data storestoring, for each of a plurality of content items, a bid specifying anamount that a content sponsor has specified for distribution of thecontent item; one or more data processing apparatus that interact withthe data store and execute instructions that cause the one or more dataprocessing apparatus to perform operations comprising: receiving acontent item request; identifying, based on information included in thecontent item request, a content item that is eligible to be provided inresponse to the content item request and is associated with a bidspecifying an amount that a content sponsor has specified fordistribution of the content item; determining that the identifiedcontent item includes two or more different interactive elements thateach cause different actions to be initiated in response to userinteraction with the different interactive elements; identifying, foreach different action, a probability that the action will be invokedthrough user interaction with an interactive element of the content itemand a bid modifier for the interaction; determining an auction score forthe content item based on a function of the bid, the bid modifier foreach different interaction, and the probability that each interactionwill be invoked; selecting, based on the auction score, the content itemto be provided in response to the content item request; and outputtingdata that cause presentation of the selected content item at a userdevice.
 15. The system of claim 14, wherein the instructions cause theone or more data processing apparatus to perform operations furthercomprising: determining a baseline cost for the content item based on anext highest auction score relative to the auction score for the contentitem, the probability that each interaction will be invoked, and the bidmodifier for each interaction.
 16. The system of claim 15, wherein theinstructions cause the one or more data processing apparatus to performoperations further comprising: determining that one of the differentactions was invoked after the content item was provided for presentationat the user device; and determining a price of providing the contentitem based on the baseline cost and the bid modifier of the invokedaction.
 17. The system of claim 16, wherein determining the pricecomprises determining a product of the baseline cost and the bidmodifier.
 18. The system of claim 15, wherein determining a baselinecost comprises determining a ratio of a sum of the next highest auctionscore and an overall negative effect of providing the content itemrelative to a weighted sum of the probabilities that each of the actionswill be invoked.
 19. The system of claim 18, wherein determining theratio comprises: determining a first product of a first probability thata user will invoke a first action and the bid modifier for the firstaction; determining a second product of a second probability that a userwill invoke a second action and the bid modifier for the second action;summing the first product and the second product to obtain the weightedsum of the probabilities; summing the next highest auction score and theoverall negative effect of providing the content item to obtain amodified next highest auction score; and determining the baseline costbased on a ratio of the modified next highest auction score and theweighted sum of the probabilities.
 20. The system of claim 14, whereindetermining an auction score for the content item comprises determininga difference between an estimated benefit corresponding to distributionof the content item and an overall negative effect of providing thecontent item.